Mobile object recognizing device, mobile object recognizing method, and computer program thereof

ABSTRACT

A mobile object recognizing device comprises a camera ( 2 ) for taking time-series images, a feature point extracting unit ( 23 ) for extracting the feature points of the individual time-series images taken by the camera ( 2 ), an optical flow creating unit ( 24 ) for comparing the feature points of the time-series images between different images, to create an optical flow joining the feature points having the same pattern, and a grouping operation unit ( 25 ) for selecting that optical flow as one belonging to one mobile object, the prolonged line of which intersects with one vanishing point within a predetermined error range and in which the external ratio of a segment joining the other end point and the vanishing point with one end point of the optical flow being the externally dividing point in the extension of the optical flow. The mobile object recognizing device may further comprise an image correcting unit ( 22 ) for correcting the image taken by the camera ( 2 ), into a transparent diagram in accordance with the characteristics of the lens of the camera ( 2 ).

TECHNICAL FIELD

The present invention relates to a mobile object recognizing device, amobile object recognizing method and a computer program thereof, whichrecognize a mobile object by use of an image.

BACKGROUND ART

In recent years, for safer driving, technologies for detecting a mobileobject approaching a vehicle, in particular, other vehicle or the likeapproaching from behind the vehicle in a traveling direction, aresuggested for the purpose of notifying a driver that the other vehicleis approaching.

For example, according to a technology of Patent Document 1, a vehiclerunning diagonally behind is detected based on whether or not an opticalflow derived from a taken image conforms to an optical flowcorresponding to a traveling speed of an own vehicle sensed by a speedsensor.

Patent Document 2 discloses a technology for recognizing other vehicleeven when driving in dim conditions. According to the technology ofPatent Document 2, in terms of images of behind and/or side of an ownvehicle, a movement of a point in a diverging direction thereof from afocus of expansion is detected as an optical flow between two imagespreceding and succeeding each other by a predetermined period of time.Then, a position of the focus of expansion of the optical flow ischanged in accordance with an amount of movement calculated based on atraveling direction information and a traveling distance information ofthe own vehicle.

Patent Document 3 discloses a technology for recognizing a mobile objectin surroundings of an own vehicle based on an obtained optical flow evenwhen the vehicle is not traveling in a straight manner. According to thetechnology of Patent Document 3, the optical flow obtained from atime-series image is classified in multiple groups of the optical flowhaving continuous changes. The optical flow is corrected based on amoving direction of the own vehicle assumed from a speed sensor and ayaw rate sensor. Then, the mobile object is extracted from the group ofthe corrected optical flow and a relative speed is calculated.

-   Patent Document 1: JPH6-130076(A)-   Patent Document 2: JPH9-86314(A)-   Patent Document 3: JPH6-282655(A)

DISCLOSURE OF INVENTION

According to known technologies, a vehicle approaching from behindand/or side is recognized by use of an image, where a basic technologyinvolves extracting an optical flow from two time-series images and thendetecting the approaching vehicle according to the optical flows.

According to a technology of Patent Document 1, a mobile object isdetected based on a difference between an optical flow and a travelingspeed of an own vehicle. On an image surface, the traveling speed of theown vehicle corresponds to the optical flow of a background. The opticalflow that differs from the optical flow of the background is recognizedas the optical flow of a mobile object. However, actions are not takenfor each mobile object because the optical flow is not grouped.

According to a technology of Patent Document 2, a mobile object isdetected from a direction of an optical flow relative to the focus ofexpansion. However, the optical flow is an apparent flow (optical flowon an image surface) and is not grouped.

According to a technology of Patent Document 3, an optical flow having acontinuous change is classified into multiple groups. However, groupingbased only on the continuity of the change is not always possiblebecause a geometric size of the optical flow is different depending onits position on an image. When a wide-angle lens is employed, inparticular, it is difficult to group even the optical flows belonging toone mobile object because they diverge greatly when approaching, andthus a direction and the size thereof become different, resulting inwrong grouping.

In light of the above, it is an object of the present invention toprovide a mobile object recognizing device that correctly detects amobile object from an image.

To achieve the above-described object, a mobile object recognizingdevice according to a first aspect of the present invention includes animage-taking means taking images in time series, a feature pointextracting means extracting a feature point of each image taken in timeseries by the image-taking means, an optical flow creating meanscreating an optical flow corresponding to a vector formed by connectingone feature point in one image with another feature point in anotherimage, the connected two feature points being identified to have thesame pattern by comparing the feature points with one another betweendifferent images, and a grouping means selecting and grouping theoptical flows created by the optical flow creating means which belong toa particular mobile object, wherein each of the selected optical flowsis defined as having an extended line intersecting at one focus ofexpansion within a predetermined error range, and having an equalexternal ratio of each line segment connecting one of the end points ofthe optical flow and the focus of expansion within a predetermined errorrange when the other one of the end points of the optical flow is anexternally dividing point on the extended line.

Preferably, the mobile object recognizing device according to the firstaspect of the present invention is characterized by including an imagecorrecting means correcting the image taken by the image-taking means inaccordance with characteristics of a lens of the image-taking means toobtain a perspective image, wherein the feature point extracting meansextracts the feature point of the image corrected by the imagecorrecting means.

Further, the mobile object recognizing device according to the firstaspect of the present invention may include a distance calculating meanscalculating a distance from the image-taking means to the mobile objectby using the optical flows grouped by the grouping means.

The mobile object recognizing device according to a second aspect of thepresent invention is characterized by including the image-taking meanstaking the images in time series, the feature point extracting meansextracting the feature point of each image taken in time series by theimage-taking means, the optical flow creating means creating the opticalflow corresponding to the vector formed by connecting one feature pointin one image with another feature point in another image, the connectedtwo feature points being identified to have the same pattern bycomparing the feature points with one another between the differentimages, a distance detecting means detecting a distance from theimage-taking means to at least one feature point extracted by thefeature point extracting means, a displacement calculating meanscalculating a displacement of the feature point in each optical flowbased on the distance to the feature point detected by the distancedetecting means and the optical flow including the feature point, and anequal displacement point selecting means selecting the feature pointshaving an equal displacement calculated by the displacement calculatingmeans as the feature points belonging to one mobile object.

Preferably, the mobile object recognizing device according to the secondaspect of the present invention is characterized by including the imagecorrecting means correcting the image taken by the image-taking means inaccordance with the characteristics of the lens of the image-takingmeans to obtain the perspective image, wherein the feature pointextracting means extracts the feature point of the image corrected bythe image correcting means.

Further, the mobile object recognizing device according to the secondaspect of the present invention may include the grouping means selectingand grouping the optical flows created by the optical flow creatingmeans which belong to the particular mobile object, wherein each of theselected optical flows is defined as having the extended lineintersecting at one focus of expansion within the predetermined errorrange, and having the equal external ratio of each line segmentconnecting one of the end points of the optical flow and the focus ofexpansion within the predetermined error range when the other one of theend points of the optical flow is the externally dividing point on theextended line, and the distance calculating means calculating thedistance from the image-taking means to the mobile object by using theoptical flows grouped by the grouping means, wherein the distancecalculated by the distance calculating means may be the distance to thefeature point.

The distance detecting means may be structured so as to measure thedistance to the mobile object by a physical means employing light, anelectromagnetic wave, a sound wave or the like.

The mobile object recognizing device according to a third aspect of thepresent invention is characterized by including an image-taking stepinputting the images in time series, a feature point extracting stepextracting the feature point of each image in time series inputted inthe image-taking step, an optical flow creating step creating theoptical flow corresponding to the vector formed by connecting onefeature point in one image with another feature point in another image,the connected two feature points being identified to have the samepattern by comparing the feature points with one another between thedifferent images, and a grouping step selecting and grouping the opticalflows, wherein each of the selected optical flows is defined as havingthe extended line intersecting at one focus of expansion within thepredetermined error range, and having the equal external ratio of eachline segment connecting one of the end points of the optical flow andthe focus of expansion within the predetermined error range when theother one of the end points of the optical flow is the externallydividing point on the extended line.

Preferably, the mobile object recognizing device according to the thirdaspect of the present invention is characterized by including an imagecorrecting step correcting the image inputted in the image-taking stepin accordance with the characteristics of the lens of the image-takingmeans to obtain the perspective image, wherein the feature pointextracting step extracts the feature point of the image corrected in theimage correcting step.

The mobile object recognizing device according to a forth aspect of thepresent invention is characterized by causing a computer to function asthe image-inputting means inputting the images in time series from theimage-taking means, the feature point extracting means extracting thefeature points of each image taken in time series by the image-inputtingmeans, the optical flow creating means creating the optical flowcorresponding to the vector formed by connecting one feature point inone image with another feature point in another image, the connected twofeature points being identified to change positions thereof on the imageand have the same pattern by comparing the feature points with oneanother between the different images, and the grouping means selectingand grouping the optical flows created by the optical flow creatingmeans which belong to the particular mobile object, wherein each of theselected optical flows is defined as having the extended lineintersecting at one focus of expansion within the predetermined errorrange, and having the equal external ratio of each line segmentconnecting one of the end points of the optical flow and the focus ofexpansion within the predetermined error range when the other one of theend points of the optical flow is the externally dividing point on theextended line.

According to the mobile object recognizing device of the presentinvention, the mobile object is detected by correctly grouping theoptical flows of the feature points extracted from the images in timeseries.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a block diagram showing a theoretical structure of a mobileobject recognizing device according to a first embodiment of the presentinvention;

FIG. 2 is a block diagram illustrating an example of a physicalstructure of the mobile object recognizing device;

FIG. 3 is a diagram describing optical flows;

FIG. 4 is a diagram showing an example of an installation position of acamera on a vehicle;

FIG. 5 is an example of an image taken by a rear monitor camera having awide-range lens;

FIG. 6 is an example of an image where a distortion of the image shownin FIG. 5 is corrected;

FIG. 7 shows an image obtained by normalizing the image shown in FIG. 6;

FIG. 8 shows an example of the optical flow;

FIG. 9 shows graphs plotted with X elements (a) and Y elements (b) of alinear equation where an external ratio of the optical flow and a focusof expansion is transformed;

FIG. 10 a diagram on which world coordinates of the optical flows areprojected on an XY plane;

FIG. 11 is a flow chart showing an example of an operation ofrecognition of the mobile object;

FIG. 12 is a block diagram showing a theoretical structure of the mobileobject recognizing device according to a second embodiment; and

FIG. 13 is a flow chart showing an example of the operation of therecognition of the mobile object of the mobile object recognizing deviceaccording to the second embodiment.

EXPLANATION OF REFERENCE NUMERALS

-   1 mobile object recognizing device-   2 camera (image-taking means)-   3 distance measuring device (distance detecting means)-   5 data holding unit-   7 display device-   10 internal bus-   11 control unit (feature point extracting means, optical flow    creating means, grouping means, displacement calculating means,    equal displacement point selecting means, image correcting means)-   12 main memory unit-   13 external memory unit-   14 operating unit-   15 screen display unit-   16 transmitting and receiving unit-   21 image input unit-   22 image correcting unit (image correcting means)-   23 feature point extracting unit (feature point extracting means)-   24 optical flow creating unit (optical flow creating means)-   25 grouping operation unit (grouping means)-   26 mobile object deciding unit-   27 display processing unit-   28 data acquisition unit-   29 equal displacement point extracting and processing unit    (displacement calculating means, equal displacement point selecting    means)-   51 collected time-series image data-   52 corrected time-series image data-   53 feature point data-   54 optical flow data-   55 grouping data-   56 distance data

MODE FOR CARRYING OUT THE INVENTION

Embodiments of the present invention will be described hereunder withreference to the accompanying drawing figures. In the drawing figures,the same reference numerals designate the same or corresponding portionsor procedures, and therefore, description thereof is not repeated. As anembodiment of a mobile object recognizing device according to thepresent invention, a mobile object recognizing device of a vehicle willbe described hereunder with reference to the drawing figures.

(First embodiment) FIG. 1 is a block diagram showing a theoreticalstructure of the mobile object recognizing device according to a firstembodiment of the present invention. The mobile object recognizingdevice 1 is constituted by a camera 2, an image input unit 21, an imagecorrecting unit 22, a feature point extracting unit 23, an optical flowcreating unit 24, a grouping operation unit 25, a data holding unit 5, amobile object deciding unit 26, a display processing unit 27, a displaydevice 7 and so forth. Collected time-series image data 51, correctedtime-series image data 52, feature point data 53, optical flow data 54and grouping data 55 are stored and held in the data holding unit 5. Themobile object recognizing device 1 recognizes whether or not there areany mobile objects in surroundings of an own vehicle and the type ofmobile object. In particular, the mobile object recognizing device 1detects other vehicle or the like approaching from behind the ownvehicle in a traveling direction of the own vehicle.

Here, an optical flow and a grouping method thereof will be described.FIG. 3 is a diagram describing the optical flow. FIG. 3 schematicallyshows a situation where the other vehicle is approaching, as the mobileobject, toward the camera 2. A feature point in a current image II₁ isx1 i (I=1 . . . n). The feature point that is in a past image II₂ andcorresponds to x1 i is x2 i. When the feature point x2 i of the pastimage II₂ is superimposed on the current image II₁, a vector from x2 ito x1 i is called the optical flow.

In case that the image is a perspective image, when the optical flow ofthe feature point of an object moving parallel to a coordinate systemfixed to the camera 2 (camera coordinate system), i.e., moving straighttoward the camera coordinate, is extended, the extended optical flowintersects at a point on the image (FIG. 3). When a straight line isdrawn from each feature point, the straight lines are parallel to eachother with respect to the camera coordinate, therefore the straightlines intersect at the point when seen in the perspective method. Thispoint is referred to as a focus of expansion (FOE). If differentindividual objects are moving but not in a parallel manner to eachother, the optical flows of these objects intersect at the differentfocuses of expansion.

For the optical flows of one object moving parallel to the cameracoordinate system, the following relationship is established when theimage is the perspective image:

$\begin{matrix}{\frac{{x\; 1i} - {x\; 2i}}{{x\; 1i} - {x\; f\; o\; e}} = C} & \lbrack {{Mathematical}\mspace{14mu}{expression}\mspace{14mu} 1} \rbrack\end{matrix}$Here, x1 i is a coordinate on the image corresponding to the featurepoint on the current image II1, x2 i is a coordinate on the imagecorresponding to the feature point on the past image II2, xfoe is acoordinate on the image corresponding to the focus of expansion, and Cis a constant.

That is, when the feature point x1 i of the current image, whichcorresponds to one end point of the optical flow, is an externallydividing point, an external ratio of a line segment connecting thefeature point x2 i of the past image (the other end point of the opticalflow) and the focus of expansion xfoe is constant within one objectparallelly moving. Therefore, the optical flows which intersect at onefocus of expansion and whose external ratios are equal to one anotherare grouped as the optical flows belonging to the one object. Instead ofthe above mentioned external ratio, for example, when the focus ofexpansion is defined as the externally dividing point, the externalratios of the optical flows within one object are constant.

A function of each portion of the mobile object recognizing device 1 isdescribed in FIG. 1. The camera 2 takes the image and transforms theimage into digital image data. Images in time series are inputted fromthe camera 2 to the image input unit 21 at constant time intervals andsaved in the data holding unit 5 as the collected time-series image data51.

As the camera 2 taking the image of the mobile object, the camera 2 fora rear monitor of a vehicle 4 as shown in FIG. 4 may be utilized. Byutilizing the camera 2 for the rear monitor, there is no need to newlyprovide the camera 2, and thus the mobile object recognizing device 1 isconstituted at a low cost.

Since a rear monitor camera of a parking assist system employs awide-angle lens, the periphery of the image is distorted and a limit ofresolution is low in a far field of the image. In addition, the rearmonitor camera is mounted on a low position. These conditions areadverse to processing the image in an actual world coordinate system,thereby making it difficult to recognize an approaching mobile object.

Hence, distortion of the taken image is corrected. The image correctingunit 22 shown in FIG. 1 corrects the distortion of the taken image inaccordance with characteristics of the lens and transforms into theperspective image. The image correcting unit 22 saves the correctedimage in the data holding unit 5 as the corrected time-series imagedata.

A relationship between a coordinate (x, y) on the perspective image anda corresponding coordinate (x′, y′) on a pre-corrected image isexpressed by the following mathematical expression:

$\begin{matrix}{{{r = \sqrt{( {x - x_{0}} )^{2} + ( {y - y_{0}} )^{2}}},{r^{\prime} = {f_{n}(r)}}}{{x^{\prime} = {{{r^{\prime}/r} \cdot ( {x - x_{0}} )} + x_{0}^{\prime}}},{y^{\prime} = {{{r^{\prime}/r} \cdot ( {y - y_{0}} )} + y_{0}^{\prime}}}}} & \lbrack {{Mathematical}\mspace{14mu}{expression}\mspace{14mu} 2} \rbrack\end{matrix}$Here, (x0, y0) is a central coordinate of the distortion of thecorrected image, (x′0, y′0) is a central coordinate of the pre-correctedimage. The fn(r) is a distortion characteristics function of the cameralens.

Further, the image may be transformed into an image in case that thecamera is oriented horizontally. A transformation for changing theorientation of the camera 2 is expressed by the following mathematicalexpression when a coordinate before the transformation is (x′, y′) and acoordinate after the transformation is (x′, y′).

[Mathematical  expression  3] ${x^{''} = {{\frac{{h_{11}( {x^{\prime} - x_{0}^{\prime}} )} + {h_{12}( {y^{\prime} - y_{0}^{\prime}} )} + {h_{13}z} + h_{14}}{{h_{31}( {x^{\prime} - x_{0}^{\prime}} )} + {h_{32}( {y^{\prime} - y_{0}^{\prime}} )} + {h_{33}z} + h_{34}} \cdot \frac{z}{\cos\;\beta}} + x_{0}^{''}}},{y^{''} = {{\frac{{h_{21}( {x^{\prime} - x_{0}^{\prime}} )} + {h_{22}( {y^{\prime} - y_{0}^{\prime}} )} + {h_{23}z} + h_{24}}{{h_{31}( {x^{\prime} - x_{0}^{\prime}} )} + {h_{32}( {y^{\prime} - y_{0}^{\prime}} )} + {h_{33}z} + h_{34}} \cdot \frac{z}{\cos\;\beta}} + y_{0}^{''} - {e\; 0}}}$Here, β expresses a tilt angle of the camera 2, z expresses a scalefactor, e0 expresses a distance between the focus of expansion and thecenter of the image before the image is transformed, and H=[hij]expresses an image transformation matrix.

FIG. 5 is an example of the image taken by the rear monitor camerahaving the wide-range lens. FIG. 6 is an example of the image where thedistortion of the image shown in FIG. 5 is corrected. In FIG. 5, whitelines on a road surface are curved, while the white lines on a straightroad are corrected to be straight in the corrected image of FIG. 6. FIG.7 shows the image obtained by normalizing the image shown in FIG. 6. Thenormalized image is obtained by performing the transformation into theimage taken by the camera 2 that is horizontally oriented. In FIG. 6,vertical lines converge at one point located in a lower portion, whilethe vertical lines are parallel to each other in FIG. 7 where theorientation of the camera 2 is corrected to be horizontal.

Next, the feature points are extracted from the corrected time-seriesimage data for conducting comparison between the images. The featurepoint extracting unit 23 extracts the feature points from the correctedtime-series image data 52 and save the feature points in the dataholding unit 5 as the feature point data 53. The optical flow creatingunit 24 compares the feature points between the images and creates theoptical flow from the vector connecting the feature points having highcorrelation with regards to their pattern. The optical flow creatingunit 24 saves the created optical flow in the data holding unit 5 as theoptical flow data 54. Methods for extracting the feature points andcreating the optical flow will be described hereunder.

A moving vector is calculated in order to detect an approaching objectin a search area obtained by way of a background subtraction. Althoughthere are several conceivable methods for calculating the moving vector,template matching may be applied, which is relatively resistant toenvironmental changes. To perform the template matching, it is importantto select an image block that is easy to be tracked. For example, acalculation method for a KLT feature point may be applied.

When a moment matrix having an intensive gradient ∇I=[Ix, Iy]T is asfollows,

$\begin{matrix}{{M( {x,y} )} = \begin{bmatrix}{Ix}^{2} & {IxIy} \\{IyIx} & {Iy}^{2}\end{bmatrix}} & \lbrack {{Mathematical}\mspace{14mu}{expression}\mspace{14mu} 4} \rbrack\end{matrix}$the following mathematical expression is established.

$\begin{matrix}{{{Ix} = \frac{\partial I}{\partial x}},{{Iy} = \frac{\partial I}{\partial y}}} & \lbrack {{Mathematical}\mspace{14mu}{expression}\mspace{14mu} 5} \rbrack\end{matrix}$When a KLT feature quantity is λ2/λ1 (condition number of matrix M (x,y)=eigenvalue ratio) or the KLT feature quantity is min (λ2, λ1), theKLT calculation method is represented by the following mathematicalexpression:

[Mathematical  expression  6]$\lambda_{1} = {\frac{{Ix}^{2} + {Iy}^{2}}{2} + \sqrt{\frac{( {{Ix}^{2} - {Iy}^{2}} )^{2}}{4} + ({IxIy})^{2}}}$$\lambda_{2} = {\frac{{Ix}^{2} + {Iy}^{2}}{2} - \sqrt{\frac{( {{Ix}^{2} - {Iy}^{2}} )^{2}}{4} + ({IxIy})^{2}}}$

A template is set around the obtained feature points, and an area havinghigh correlations with the template is searched from the chronologicallynewer image. A relevant motion vector is obtained as the moving vector(optical flow). As a method of the template matching, a method of thenormalized cross-correlation (NCC) may be applied. A degree ofsimilarity (RNCC) of the normalized cross-correlation is obtained by thefollowing mathematical expression:

       [Mathematical  expression  7]$R_{N\; C\; C} = \frac{( {\sum\limits_{j = 1}^{N - 1}\;{\sum\limits_{i = 1}^{M - 1}\;( {( {{I( {i,j} )} - \overset{\_}{I}} )( {{T( {i,j} )} - \overset{\_}{T}} )} )}} )}{( \sqrt{\sum\limits_{j = 1}^{N - 1}\;{\sum\limits_{i = 1}^{M - 1}\;{( {{I( {i,j} )} - \overset{\_}{I}} )^{2} \times {\sum\limits_{j = 1}^{N - 1}\;{\sum\limits_{i = 1}^{M - 1}\;( {{T( {i,j} )} - \overset{\_}{T}} )^{2}}}}}} )}$Here, I is an intensity vector in an area of N row by M column and T isthe intensity vector in a template area of N row by M column. Bars abovethe variables represent average values in each area respectively.

Since a full search of the area requires a large amount of calculation,a limitation of an amount of movement of the mobile object is assumeddepending on the situation and only the limited area is searched toreduce a calculation load. For example, the search is not performed inany other directions than an approaching direction. Alternatively, thesearch is not performed in an area where a physical possibility isassumed to be low according to an amount of movement in the past, and soforth. Still alternatively, since the object that is close to the camera2 shows a large amount of movement, a hierarchization is performed, thesearch area is narrowed by conducting the matching in a higherhierarchical level, and then a detailed search is performed in order toreduce the amount of calculation.

FIG. 8 shows an example of the optical flow. Open rectangles representthe feature points in a previous frame and open circles represent pointsin an updated frame having high correlations with the feature points inthe previous frame. Even though the optical flows belong to one mobileobject, for example, the optical flow of a radiator grill and theoptical flow of a rear wheel are different in their directions andlengths, and thus it is difficult to group them as the optical flowsbelonging to one mobile object.

Hence, as previously described, the optical flows which intersect at onefocus of expansion and whose external ratios expressed in mathematicalexpression 1 are equal to one another are grouped as the optical flowsbelonging to one object. The grouping operation unit 25 groups theoptical flow, for example, as follows.

The mathematical expression 1 includes two parameters (the focus ofexpansion and external ratio constant) and grouping conditions requireeach optical flow has the same parameters. Here, the mathematicalexpression 1 is transformed into a linear equation for two variables.dxi=C·x1i−C1  [Mathematical expression 8]Here, dxi=x1 i−x2 i, and C1=C·xfoe.

When dxi and x1 i are plotted on planes in terms of the x coordinate andthe y coordinate respectively for each optical flow, FIG. 9 is obtained.FIG. 9 (a) shows x elements of dxi for the x coordinate of x1 i. FIG. 9(b) shows y elements of dxi for the y coordinate of x1 i.

Among the points plotted as shown in FIG. 9, the points lying on astraight line allowing a certain error are grouped as the optical flowsthat belong to one and the same mobile object. Although there arevarious methods to judge whether or not the point is on the straightline, for example, the RANSAC (RANdom SAmple Consensus) may be applied.The RANSAC is an algorithm where any two points are selected andconnected with a straight line, the number of points lying within thetolerance of the straight line is counted, and the straight line havingthe greatest number of points is selected. The points on the selectedstraight line with the tolerance are grouped as one group of the opticalflows.

Aside from the points that have been classified as one group, in orderto further group the optical flows belonging to another mobile object,the points lying on a straight line allowing the certain error areselected. By repeating this, multiple mobile objects are recognized.

The grouping operation unit 25 saves data of the grouped optical flowsin the data holding unit 5 as the grouping data 55. The mobile objectdeciding unit 26 makes decisions on the mobile object based on thegrouping data 55. For example, a size of the mobile object, a vehiclemodel and so forth are decided based on a placement of the featurepoints. Further, the grouping operation unit 26 calculates a distance tothe mobile object.

Assuming that the lowest point of the optical flows is the road surface(the same plane as the one on which the own vehicle is positioned), thedistance to the mobile object is calculated by using the followingmathematical expression. Here, a distance Z is the direction cosine of adistance from the camera to the object with respect to an imagingdirection. The imaging direction is a direction of an optical axis ofthe camera, which corresponds to a horizontal direction (parallel to theroad surface) when the image is normalized in the horizontal direction.Hereinafter, the imaging direction corresponds to the horizontaldirection (parallel to the road surface). A coordinate of the featurepoint on an image plane is (u, v), the world coordinate is (X, Y, Z) anda central coordinate of the image is (cu, cv). U=u−cu and V=v−cv.

       [Mathematical  expression  9] ${Z = \frac{\begin{matrix}{{( {{\frac{f}{\delta\; v}r_{21}} - {Vr}_{31}} )( {{\frac{f}{\delta\; u}( {t_{1} - {r_{12}h}} )} - {U( {t_{3} - {r_{33}h}} )}} )} -} \\{( {{\frac{f}{\delta\; u}r_{11}} - {Ur}_{31}} )( {{\frac{f}{\delta\; v}( {t_{2} - {r_{22}h}} )} - {V( {t_{3} - {r_{32}h}} )}} )}\end{matrix}}{\begin{matrix}{{( {{\frac{f}{\delta\; v}r_{21}} - {Vr}_{31}} )( {{{- \frac{f}{\delta\; u}}r_{13}} + {Ur}_{33}} )} -} \\{( {{\frac{f}{\delta\; u}r_{11}} - {Ur}_{31}} )( {{{- \frac{f}{\delta\; v}}r_{23}} - {Vr}_{33}} )}\end{matrix}}}\mspace{11mu}$Here, f is a focal distance of the camera 2, δu and δv are physicalintervals between pixels in a lateral direction and a longitudinaldirection respectively, and h is a height of the camera from the roadsurface. R=[rij] is a rotating matrix with a coefficient of a projectivetransformation matrix and T=[tk]^(T) is a parallel motion vector withthe coefficient of the projective transformation matrix.

The display processing unit 27 displays decision results made by themobile object deciding unit 26 on the display device 7. For example, arelationship between a position of a vehicle approaching from behind andthe own vehicle is shown in a plan view. Further, the display on thedisplay device 7 may include acoustic effects such as sound or chime.

FIG. 2 is a block diagram illustrating a physical structure of themobile object recognizing device 1. As shown in FIG. 2, the mobileobject recognizing device 1 according to the present invention shown inFIG. 1 is constituted by hardware of a control unit 11, a main memoryunit 12, an external memory unit 13, an operating unit 14, a screendisplay unit 15, a transmitting and receiving unit 16, the camera 2, adistance measuring device 3, and a display device 7. All of the controlunit 11, the main memory unit 12, the external memory unit 13, theoperating unit 14, the screen display unit 15, and the transmitting andreceiving unit 16 are connected to the control unit 11 via an internalbus 10.

The control unit 11 is constituted by a CPU (Central Processing Unit)and so forth, and executes processing of the image input unit 21, theimage correcting unit 22, the feature point extracting unit 23, theoptical flow creating unit 24, the grouping operation unit 25, themobile object deciding unit 26, and the display processing unit 27 inaccordance with a program stored in the external memory unit 13. Theimage input unit 21, the image correcting unit 22, the feature pointextracting unit 23, the optical flow creating unit 24, the groupingoperation unit 25, the mobile object deciding unit 26, and the displayprocessing unit 27 are realized by the control unit 11 and the programexecuted on the control unit 11.

The main memory unit 12 is constituted by a RAM (Random-Access Memory)and so forth, and is used as a working space of the control unit 11. Thedata holding unit 5 is stored and held, as a structure of a memory area,in a part of the main memory unit 12.

The external memory unit 13 is constituted by a nonvolatile memoryincluding a flash memory, a hard disc, a DVD (Digital Versatile Disc), aDVD-RAM (Digital Versatile Disc Random-Access Memory), a DVD-RW (DigitalVersatile Disc ReWritable) and stores in advance the program for causingthe control unit 11 to execute the above-mentioned processing. Inaddition, the external memory unit 13 provides the control unit 11 withdata of the program and stores data provided by the control unit 11according to instructions of the control unit 11. For example, thetime-series image data may be stored in the external memory unit 13.

The operating unit 14 is provided with pointing devices and so forthincluding a key switch, a jog dial, a keyboard, a mouse, and aninterface device connecting them to the internal bus 10. Conditions forrecognizing and deciding the mobile object, a command to startrecognizing the mobile object and so forth are inputted via theoperating unit 14, and then provided to the control unit 11.

The display device 7 is constituted by a CRT (Cathode Ray Tube), an LCD(Liquid Crystal Display) or the like, and displays the collectedtime-series image data 51, the corrected time-series image data 52, thedecision results on the mobile object and so forth in response to theinstruction of the control unit 11 according to the command inputted viathe operating unit 14. The screen display unit 15 transforms the imagedata to be displayed on the display device 7 into a signal that actuatesthe display device 7.

The display device 7 may be provided with a speaker, a buzzer or thelike. In that case, the screen display unit 15 causes the acousticeffects such as the sound or the chime to be outputted from the displaydevice 7.

The camera 2, for example, transforms the image formed on the lens intoan electric signal by use of a CCD or the like and outputs the imagedata that is digitalized pixel-by-pixel. The distance measuring device 3corresponds, for example, to a millimeter wave radar, a laser rangefinder, an ultrasonic distance meter or the like and outputs thedistance to the mobile object as an electric signal. The distancemeasuring device 3 is not employed in the first embodiment. The distancemeasuring device 3 is employed to measure the distance to the object ina second embodiment mentioned below.

The transmitting and receiving unit 16 is constituted by a modem or anetwork terminator, and a serial interface or a LAN (Local Area Network)interface connected thereto. Time-series data is inputted to the controlunit 11 from the camera 2 or the distance measuring device 3 via thetransmitting and receiving unit 16.

Next, an operation of the mobile object recognizing device 1 will bedescribed. As described above, the mobile object recognizing device 1 isoperated by the control unit 11 in cooperation with the main memory unit12, the external memory unit 13, the operating unit 14, the screendisplay unit 15, and the transmitting and receiving unit 16.

FIG. 11 is a flow chart showing an example of the operation ofrecognizing the mobile object. The images in time series are inputted tothe control unit 11 from the camera 2 via the transmitting and receivingunit 16 (Step A1). Then, as previously described, the image is correctedin accordance with the characteristics of the lens of the camera 2 (StepA2). Further, the image may be normalized so as to be the image in thehorizontal direction by correcting the tilt angle of the camera 2.

The control unit 11 extracts the feature points of the corrected imageby, for example, an extraction method for the KLT feature pointpreviously described (Step A3). The control unit 11 compares theextracted feature points between the images in time series and createsthe optical flows by applying the template matching (Step A4).

Next, the optical flows are grouped (Step A5). As previously described,among the points plotted as shown in FIG. 9, the points on the straightline allowing the certain error are grouped as the optical flows thatbelong to one and the same mobile object. To judge whether or not thepoint is on the straight line, for example, the RANSAC is applied. Then,the mobile object is decided based on the patterns of the feature pointsbelonging to the grouped optical flows (Step A6). Further, the distanceto the mobile object may be calculated in a manner shown in themathematical expression 9, assuming that the lowest point of the opticalflows is the road surface (the same plane as the one on which the ownvehicle is positioned).

When the mobile object is recognized (Yes in Step A7), the control unit11 causes the information to be displayed on the display device 7 viathe screen display unit 15 (Step A8). For example, the relationshipbetween the position of the vehicle approaching from behind and that ofthe own vehicle is shown in the plan view. Alternatively, the displaydevice 7 may notify a driver by means of the acoustic effects such asthe sound or the chime. Then, the operation returns to the input of theimages in time series (Step A1) and the recognition of the mobile objectis repeated. When no mobile object is recognized (No in Step A7), nodisplay takes place and the recognition of the mobile object is repeated(Step A1).

According to the mobile object recognizing device 1 of the presentinvention, the optical flows created from the series-image data arecorrectly grouped for each mobile object. As a result, the mobile objectis correctly recognized. In addition, the image is corrected inaccordance with the characteristics of the lens of the camera 2, therebythe optical flows created from the corrected image data are correctlygrouped for each mobile object even when a rear camera employing thewide-range lens is used.

(Second embodiment) Next, the mobile object recognizing device 1 will bedescribed, where a coordinate of the feature point of the optical flowis calculated and the feature points whose displacements are equalbetween two images are grouped as the feature points that belong to onemobile object. FIG. 12 is a block diagram showing a theoreticalstructure of the mobile object recognizing device 1 according to thesecond embodiment of the present invention.

The mobile object recognizing device 1 is constituted by the camera 2,the distance measuring device 3, a data acquisition unit 28, the imagecorrecting unit 22, the feature point extracting unit 23, the opticalflow creating unit 24, an equal displacement point extracting andprocessing unit 29, the data holding unit 5, the mobile object decidingunit 26, the display processing unit 27, the display device 7 and soforth. The collected time-series image data 51, distance data 56, thecorrected time-series image data 52, the feature point data 53, theoptical flow data 54 and the grouping data 55 are stored and held in thedata holding unit 5.

The camera 2, the image correcting unit 22, the feature point extractingunit 23, the optical flow creating unit 24, the mobile object decidingunit 26, the display processing unit 27 and the display device 7 areidentical to those of the first embodiment.

The distance measuring device 3 corresponds to measuring equipmentmeasuring a distance by using a physical method, including, for example,the millimeter wave radar, the laser range finder, the ultrasonicdistance meter. The distance measuring device 3 measures the distance tothe object positioned in a direction of which image is taken by thecamera 2. The data acquisition unit 28 acquires the time-series imagefrom the camera 2 and saves the image as the collected time-series imagedata 51 in the data holding unit 5. In addition, the data acquisitionunit 28 acquires the data of the distance measuring device 3 and savesthe data as the distance data 56 in the data holding unit 5.

The equal displacement point extracting and processing unit 29calculates the displacement of the feature point in the optical flowbased on the distance to the feature point and based on the optical flowincluding the feature point. In addition, the equal displacement pointextracting and processing unit 29 groups the feature points having thesame calculated displacements as the feature points belonging to onemobile object. Then, the equal displacement point extracting andprocessing unit 29 saves data of the grouped optical flows as thegrouping data 55 in the data holding unit 5.

From the distance Z to the feature point corresponding to the end pointof the optical flow, the world coordinate (X, Y) of the feature point iscalculated by the following mathematical expression. Here, the distanceZ is the direction cosine of the distance from the camera to the objectwith respect to the imaging direction. The imaging direction is thedirection of the optical axis of the camera, which corresponds to thehorizontal direction (parallel to the road surface) when the image isnormalized in the horizontal direction. Hereinafter, the imagingdirection corresponds to the horizontal direction (parallel to the roadsurface). The coordinate of the feature point on the image plane is (u,v), the world coordinate is (X, Y, Z) and the central coordinate of theimage is (cu, cv). (Xnum, Ynum) is the world coordinate of a startingpoint of the optical flow. The value previously calculated may beapplied as the world coordinate of the starting point.

[Mathematical  expression  10]X_(num) = (c_(u) − u)(r₃₂r₂₃ − r₂₂r₃₃)S_(v)Z + (c_(v) − v)(r₁₂r₃₃ − r₃₂r₁₃)S_(u)Z + (r₁₂r₁₃ − r₂₂r₃₃)S_(u)S_(v)Z + (c_(u) − u)(r₃₂t₂ − r₂₂t₃)S_(v) + (c_(v) − v)(r₁₂t₃ − r₃₂t₁)S_(u) + (r₁₂t₂ − r₂₂t₁)S_(u)S_(v)$X = {{\frac{X_{num}}{\begin{matrix}{{( {c_{u} - u} )( {{r_{31}r_{22}} - {r_{32}r_{21}}} )S_{v}} +} \\{{( {c_{v} - v} )( {{r_{11}r_{32}} - {r_{12}r_{31}}} )S_{u}} + {( {{r_{11}r_{22}} - {r_{12}r_{21}}} )S_{u}S_{v}}}\end{matrix}}Y_{num}} = {{( {c_{u} - u} )( {{r_{21}r_{33}} - {r_{31}r_{23}}} )S_{v}Z} + {( {c_{v} - v} )( {{r_{31}r_{13}} - {r_{11}r_{33}}} )S_{u}Z} + {( {{r_{21}r_{13}} - {r_{11}r_{23}}} )S_{u}S_{v}Z} + {( {c_{u} - u} )( {{r_{21}t_{3}} - {r_{31}t_{2}}} )S_{v}} + {( {c_{v} - v} )( {{r_{31}t_{1}} - {r_{11}t_{3}}} )S_{u}} + {( {{r_{21}t_{1}} - {r_{11}t_{2}}} )S_{u}S_{v}}}}$$Y = \frac{Y_{num}}{\begin{matrix}{{( {c_{u} - u} )( {{r_{31}r_{22}} - {r_{32}r_{21}}} )S_{v}} +} \\{{( {c_{v} - v} )( {{r_{11}r_{32}} - {r_{12}r_{31}}} )S_{u}} + {( {{r_{11}r_{22}} - {r_{12}r_{21}}} )S_{u}S_{v}}}\end{matrix}}$Here, Su and Sv are the scale factors, R=[rij] is the rotating matrixwith the coefficient of the projective transformation matrix andT=[tk]^(T) is the parallel motion vector with the coefficient of theprojective transformation matrix. That is, when θ Pan, θ Tilt and θ Rollare mounting angles of the camera and T_(X), T_(Y) and T_(Z) aremounting positions of the camera, the coefficient of the projectivetransformation matrix is expressed by the following mathematicalexpression:

$\begin{pmatrix}r_{11} & r_{12} & r_{13} & t_{1} \\r_{21} & r_{22} & r_{23} & t_{2} \\r_{31} & r_{32} & r_{33} & t_{3} \\0 & 0 & 0 & 1\end{pmatrix} = {\begin{pmatrix}{\cos\;\theta_{Pan}} & 0 & {\sin\;\theta_{Pan}} & 0 \\0 & 1 & 0 & 0 \\{{- \sin}\;\theta_{Pan}} & 0 & {\cos\;\theta_{Pan}} & 0 \\0 & 0 & 0 & 1\end{pmatrix}\begin{pmatrix}1 & 0 & 0 & 0 \\0 & {\cos\;\theta_{Tilt}} & {{- \sin}\;\theta_{Tilt}} & 0 \\0 & {\sin\;\theta_{Tilt}} & {\cos\;\theta_{Tilt}} & 0 \\0 & 0 & 0 & 1\end{pmatrix}\begin{pmatrix}{\cos\;\theta_{Roll}} & {{- \sin}\;\theta_{Roll}} & 0 & 0 \\{\sin\;\theta_{Roll}} & {\cos\;\theta_{Roll}} & 0 & 0 \\0 & 0 & 1 & 0 \\0 & 0 & 0 & 1\end{pmatrix}\begin{pmatrix}1 & 0 & 0 & T_{X} \\0 & 1 & 0 & T_{Y} \\0 & 0 & 1 & T_{Z} \\0 & 0 & 0 & 1\end{pmatrix}}$

With regard to the distance Z to the end point, the previouslycalculated value is used as a Z coordinate of the starting point and thevalue calculated in the mathematical expression 9 with assumption thatthe lowest point of the optical flows is the road surface is used as a Zcoordinate of an ending point. That is, the distance Z calculated in thefirst embodiment may be applied. In that case, the distance measuringdevice 3 does not have to be used.

By measuring the distance Z to each feature point and calculating theworld coordinate of each feature point, the feature points whosedisplacements (vectors) in terms of the world coordinate system areequal are grouped as the feature points that belong to the one mobileobject. For approximation purposes, distances Z to the feature points inan operation area may be represented by one value to calculate the worldcoordinate, considering the distances Z (direction cosine with respectto the imaging direction) to be the same. For approximation purposes,the grouping is performed, for example, as follows;

When the coordinates (X, Y) of the calculated end points are plotted onthe plane so that the calculated end points serve as the starting pointsand the ending points of the vectors, FIG. 10 is obtained. The opticalflows whose vectors have the identical direction and the length, thatis, the optical flows whose displacements are equal to one anotherallowing the certain error are grouped as the feature points belongingto one mobile object. Based on the grouped optical flows, the decisionsare made on the mobile body and a display processing is performed in thesame manner as described in the first embodiment.

Next, the operation of the mobile object recognizing device 1 will bedescribed, where the coordinate of the feature point of the optical flowis calculated and the feature points whose displacements are equalbetween two images are grouped as the feature points that belong to onemobile object. A physical structure of the mobile object recognizingdevice 1 according to the second embodiment is shown in, for example,FIG. 2. FIG. 13 is a flow chart showing an example of the operation forrecognizing the mobile object of the mobile object recognizing device 1according to the second embodiment.

The images in time series are inputted to the control unit 11 from thecamera 2 via the transmitting and receiving unit 16 (Step B1). Thedistance data 56 is inputted to the control unit 11 from the distancemeasuring device 3 (Step B2). When the distance Z takes the valuecalculated in the mathematical expression 9 with the assumption that thelowest point of the optical flows is the road surface, the previousvalue is applied. As previously described, the control unit 11 correctsthe image in accordance with the characteristics of the lens of thecamera 2 (Step B3). Further, the image may be normalized in thehorizontal direction by correcting the tilt angle of the camera 2.

The control unit 11 extracts the feature points of the corrected imageby, for example, the previously-described KLT feature point extractionmethod (Step B4). The control unit 11 compares the extracted featurepoints between the images in time series and creates the optical flow byapplying the previously-described template matching (Step B5).

Next, the world coordinate of the feature point is calculated based onthe optical flow and the distance Z (Step B6). When the distance Z takesthe value calculated in the mathematical expression 9 with theassumption that the lowest point of the optical flows is the roadsurface, the distance Z is calculated from the optical flow of thefeature point that was the lowest point in the previous case. Then,points with equal displacement are grouped (Step B7). As previouslydescribed, among the vectors plotted as in FIG. 10, the vectors havingthe same directions and sizes allowing the certain error are grouped asthe optical flows that belong to one and the same mobile object.

The mobile object is decided based on the patterns of the feature pointsbelonging to the grouped optical flows (Step B8). Further, the distanceto the mobile object may be calculated in the manner shown in themathematical expression 9 with the assumption that the lowest point ofthe optical flows is the road surface (the same plane as the one onwhich the own vehicle is positioned).

When the mobile object is recognized (Yes in Step B9), the control unit11 causes the information to be displayed on the display device 7 viathe screen display unit 15 (Step B10). For example, the relationshipbetween the position of the vehicle approaching from behind and that ofthe own vehicle is shown in the plan view. Alternatively, the displaydevice 7 may notify the driver by means of the acoustic effects such asthe sound or the chime. Then, the operation returns to the input of theimages in time series (Step B1) and the recognition of the mobile objectis repeated. When no mobile object is recognized (No in Step B9), nodisplay takes place and the recognition of the mobile object is repeated(Step B1).

According to the mobile object recognizing device 1 of the presentinvention, the optical flows created from the series-image data arecorrectly grouped for each mobile object. As a result, the mobile objectis correctly recognized. In addition, the mobile object recognizingdevice 1 corrects the image in accordance with the characteristics ofthe lens of the camera 2, thereby the optical flows created from thecorrected image data are correctly grouped for each mobile object evenwhen the rear camera employing the wide-range lens is used.

The previously described structure of the hardware and flow charts areexamples and may be arbitrarily modified and revised.

A leading part of the mobile object recognizing device 1 which isconstituted by the control unit 11, the main memory unit 12, theexternal memory unit 13, the operating unit 14, the screen display unit15, the transmitting and receiving unit 16, the internal bus 10 and soforth and which performs the processing of the mobile object recognizingdevice 1 may be achieved by use of a conventional computer systeminstead of a dedicated system. For example, the mobile objectrecognizing device 1 executing the previously-described processing maybe so structured that the computer programs for executing thepreviously-described operations are stored in a memory medium readableby the computer (a flexible disc, CD-ROM, DVD-Rom or the like) andprovided, and then installed on the computer. Alternatively, the mobileobject recognizing device 1 may be so structured that the computerprograms are stored in a memory device provided in a server device on acommunication network including the Internet, and, for example,downloaded by the conventional computer system.

Still alternatively, in case that the mobile object recognizing device 1causes the functions thereof to be shared between an OS (operatingsystem) and an application program, or to be cooperated by the OS andthe application program, only the application program part may be storedin the memory medium or the memory device.

Still alternatively, the computer program may be superimposed on acarrier wave and distributed via the communication network. For example,the previously-described computer program may be posted on a bulletinboard on the communication network (BBS, Bulletin Board System) anddistributed via the network. Then, the structure may be available wherethe computer program is run and executed like other application programsunder control of the OS, thereby performing the previously-describedprocessing.

The embodiments disclosed this time should be regarded asexemplifications in every aspect and should not be construed in alimited manner. It is intended that the present invention should beindicated by the scope of claims and not by the above-describedexplanation, and that all the modifications are included within thescope of claims, and within a meaning and a range of equivalence.

The present application is based on Japanese patent application No.2006-275387 filed on Oct. 6, 2006. The whole of the specification, thescope of claims for patent application, and the drawings of the Japanesepatent application No. 2006-27538 is included in the presentspecification as reference.

INDUSTRIAL APPLICABILITY

The present invention is applicable to a mobile object recognizingdevice for correctly grouping optical flows of feature points extractedfrom images in time series for each mobile object and detecting themobile object.

1. A mobile object recognizing device comprising; an image-taking unitthat is mounted on a vehicle and takes images in time series; a featurepoint extracting means extracting a feature point of each image taken intime series by the image-taking unit; an optical flow creating unit thatcreates optical flows corresponding to a vector formed by connecting onefeature point in one image with another feature point in another image,the connected two feature points being identified to have the samepattern by comparing the feature points with one another betweendifferent images; and a grouping means selecting and grouping theoptical flows created by the optical flow creating unit which belong toa particular mobile object, wherein each of the selected optical flowsis defined as having an extended line intersecting at one focus ofexpansion within a predetermined error range, and having an equalexternal ratio of each line segment connecting one of the end points ofthe optical flow and the focus of expansion within a predetermined errorrange when the other one of the end points of the optical flow is anexternally dividing point on the extended line.
 2. The mobile objectrecognizing device according to claim 1, further comprising: an imagecorrecting means correcting the image taken by the image-taking unit inaccordance with characteristics of a lens of the image-taking unit toobtain a perspective image, wherein the feature point extracting meansextracts the feature point of the image corrected by the imagecorrecting means.
 3. The mobile object recognizing device according toclaim 1, further comprising: a distance calculating means calculating adistance from the image-taking unit to the mobile object by using theoptical flows grouped by the grouping means.
 4. A mobile objectrecognizing device comprising: an image-taking unit that is mounted on avehicle and takes images in time series; a feature point extractingmeans extracting a feature point of each image taken in time series bythe image-taking unit; an optical flow creating unit that createsoptical flows corresponding to a vector formed by connecting one featurepoint in one image with another feature point in another image, theconnected two feature points being identified to have the same patternby comparing the feature points with one another between differentimages; a distance detecting means detecting a distance from theimage-taking unit to at least one feature point extracted by the featurepoint extracting means; a displacement calculating means calculating adisplacement of the feature point in each optical flow based on thedistance to the feature point detected by the distance detecting meansand the optical flow including the feature point; an equal displacementpoint selecting means selecting the feature points having an equaldisplacement calculated by the displacement calculating means as thefeature points belonging to one mobile object; a grouping meansselecting and grouping the optical flows created by the optical flowcreating unit which belong to a particular mobile object, wherein eachof the selected optical flows is defined as having an extended lineintersecting at one focus of expansion within a predetermined errorrange, and having an equal external ratio of each line segmentconnecting one of the end points of the optical flow and the focus ofexpansion within a predetermined error range when the other one of theend points of the optical flow is an externally dividing point on theextended line; and a distance calculating means calculating a distancefrom the image-taking unit to the mobile object by using the opticalflows grouped by the grouping means, wherein the distance calculated bythe distance calculating means is the distance to the feature point. 5.The mobile object recognizing device according to claim furthercomprising an image correcting means correcting the image taken by theimage-taking unit in accordance with characteristics of a lens of theimage-taking unit to obtain a perspective image, wherein the featurepoint extracting means extracts the feature point of the image correctedby the image correcting means.
 6. The mobile object recognizing deviceaccording to claim 4, wherein the distance detecting means measures thedistance to the mobile object by a physical means.
 7. A mobile objectrecognizing method comprising: an image-taking step inputting images intime series; a feature point extracting step extracting feature point ofeach image inputted in time series in the image-taking step; an opticalflow creating step creating optical flows corresponding to a vectorformed by connecting one feature point in one image with another featurepoint in another image, the connected two feature points beingidentified to have the same pattern by comparing the feature points withone another between different images; and a grouping step selecting andgrouping the optical flows, wherein each of the selected optical flowsis defined as having an extended line intersecting at one focus ofexpansion within a predetermined error range, and having an equalexternal ratio of each line segment connecting one of the end points ofthe optical flow and the focus of expansion within a predetermined errorrange when the other one of the end points of the optical flow is anexternally dividing point on the extended line.
 8. The mobile objectrecognizing method according to claim 7, further comprising: an imagecorrecting step correcting the image inputted in the image-taking stepin accordance with characteristics of a lens of the image-taking unit toobtain a perspective image, wherein the feature point extracting stepextracts the feature point of the image corrected in the imagecorrecting step.
 9. A non-transitory computer-readable medium causing acomputer to function as: an image-inputting means inputting images intime series from an image-taking unit; a feature point extracting meansextracting a plurality of feature points of each image taken in timeseries by the image-inputting means; an optical flow creating unit thatcreates optical flows corresponding to a vector formed by connecting onefeature point in one image with another feature point in another image,the connected two feature points being identified to change positionsthereof on the image and have the same pattern by comparing the featurepoints with one another between different images; and a grouping meansselecting and grouping the optical flows created by the optical flowcreating unit which belong to a particular mobile object, wherein eachof the selected optical flows is defined as having an extended lineintersecting at one focus of expansion within a predetermined errorrange, and having an equal external ratio of each line segmentconnecting one of the end points of the optical flow and the focus ofexpansion within a predetermined error range when the other one of theend points of the optical flow is an externally dividing point on theextended line.